1. Field
The embodiments described herein relate generally to video image processing, and more particularly, to a method and apparatus for motion vector assignment.
2. Background
Low bit rate video compression is very important in many multimedia applications such as wireless video streaming and video telephony, due to the limited bandwidth resources and the variability of available bandwidth. Bandwidth adaptation video coding at low bit-rate can be accomplished by reducing the temporal resolution. In other words, instead of compressing and sending a thirty (30) frame per second (fps) bit-stream, the temporal resolution can be halved to 15 fps to reduce the transmission bit-rate. One typical approach for reducing the bandwidth is through the extraction and removal of redundant information from the encoded signal, such as removing either a portion of or the complete frame. However, the consequence of reducing temporal resolution is the introduction of temporal domain artifacts such as motion jerkiness that significantly degrades the visual quality of the decoded video.
To display the full frame rate at the receiver side, a recovery mechanism, called frame rate up conversion (FRUC), is needed to re-generate the skipped frames and to reduce temporal artifacts. Generally, FRUC is the process of video interpolation at the video decoder to increase the perceived frame rate of the reconstructed video. Many FRUC algorithms have been proposed, and can be classified into two general categories. The first category interpolates the missing frame by using a combination of received video frames without taking object motion into account. Frame repetition and frame averaging methods fit into this class. The drawbacks of these methods include the production of motion jerkiness, “ghost” images and blurring of moving objects when there is motion involved. The second category is more advanced as compared to the first category, and utilizes motion information transmitted by the encoder in a process referred to as motion compensated (frame) interpolation (MCI).
In conventional motion compensated predictive coding, each video frame is first partitioned into square blocks of picture elements (pels); such as blocks of 8 pels by 8 pels. Each block is coded, in turn, and the developed encoded sequence is transmitted over a communications channel to a decoder. During the encoding process, a determination is made as to whether or not the pels of the block have changed significantly compared with a collocated block of a previous frame. If not, an indication is sent that signifies to the decoder that it needs to merely repeat the pels of that block from the previous frame to obtain the pels for the current block. If the pels have changed since the previous frame, an attempt is made to determine the best estimate of motion that is occurring in the block. This is frequently done by a block matching motion estimation technique, where the pels of the current block are successively compared with various small shifts of the corresponding block in the previous frame. The shift that provides the best match is deemed to be the “best estimate” of the displacement in the block's image between frames, and the magnitude and direction of this shift, referred to as a “motion vector,” is selected to be sent to the decoder. Also, once the best match is found, the pels of the current block are compared with those of the “best” shifted block from the previous frame to see if there is a significant difference. If not, instead of sending the motion vector, an indication is sent to the decoder so that the pels of the shifted block from the previous frame will be repeated for the pels for the current shifted block. Such blocks are said to have been successfully motion compensated. However, if there is a significant difference between the two blocks, the difference is encoded and sent to the decoder so that the pels of the current block may be more accurately recovered.
There have been some suggestions for improving the quality of the repeated frames in order to make them more faithfully resemble the original. Thus, instead of simply repeating the pels from the previous frame as described above in conventional motion compensated predictive coding, in MCI motion vectors are used to laterally displace a block by the appropriate amount prior to display. In other words, this method creates the missing block of pels by averaging over the immediately previous and following blocks of pels that are available to the decoder. However, experimental results show that when the images of successive blocks do not represent translational motion, the reproduced image may be worse than with frame repeating. Although it has been observed that this degradation is caused by a relatively few pels that do not conform to the assumption of translational motion, putting these pels in the wrong place creates highly visible artifacts.
As illustrated in prior art FIG. 1, in MCI, a missing frame 108 is interpolated based on a reconstructed current frame 102, a stored previous frame 104, and a set of transmitted motion vectors 106. Reconstructed current frame 102 is composed of a set of non-overlapped blocks 150, 152, 154 and 156 associated with set of transmitted motion vectors 106 pointing to corresponding blocks in stored previous frame 104. Interpolated frame 108 can be constructed in either a linear combination of corresponding pixels in current and previous frames; or nonlinear operation such as a median operation.
Although MCI is more advanced, block-based MCI introduces overlapped regions, which are areas having multiple motion trajectories passing through it; and hole regions, which are areas having no motion trajectory passing through it, both of which are unwanted, in interpolated frames. As illustrated in FIG. 2, an interpolated frame 202 contains an overlapped area 206 and a hole area 204. The main reasons for the generation of these two types of unwanted areas are:                1. Moving objects are not under a rigid translational motion model.        2. The transmitted motion vectors used in MCI may not point to the true motion trajectories due to the block-based fast motion search algorithms utilized on the encoder side.        3. The covered and uncovered background in the current frame and previous frames.        
The interpolation of overlapped and hole regions is a major technical challenge in conventional block-based motion compensated approaches. Median blurring and spatial interpolation techniques have been proposed to fill these overlapped and hole regions. However, the drawbacks of these methods are the introduction of the blurring and blocking artifacts, and also an increase in the complexity of interpolation operations.
Proper motion vector assignment is important for motion compensated FRUC as it is needed for the interpolated frames in motion compensated FRUC. In addition, proper motion vector assignment is useful in any application where proper motion compensation based on motion vector assignment is needed. Accordingly, there is a need to overcome the issues noted above.